Over the last decade superpixel algorithms have become a broadly accepted and applied method for image segmentation, providing a reduction in complexity for subsequent processing tasks. Superpixel segmentation provides the advantage of switching from a rigid structure of the pixel grid of an image to a semantic description defining objects in the image, which explains its popularity in image processing and computer vision algorithms.
Research on superpixel algorithms began with a processing intensive feature grouping method proposed in [1]. Subsequently, more efficient solutions for superpixel generation were proposed, such as the simple linear iterative clustering (SLIC) method introduced in [2]. While earlier solutions focused on still images, later developments aimed at applications of superpixels for video, which require their temporal consistency. In [3] an approach achieving this demand is described, which provides traceable superpixels within video sequences.
The temporal dimension in image processing and computer vision algorithms requires the tracking of objects in video by tracking superpixels over time. This is an easy task when a macro tracking of superpixels is needed, requiring a simple superpixel-to-superpixel mapping. More often, however, image processing requires a micro tracking, describing the pixel-to-pixel correspondence between temporally adjacent superpixels, i.e. between macro tracked superpixels.
The difficulty of micro tracking arises from the shape of the superpixels deforming over time. While this does not harm the macro tracking, it definitely eliminates the possibility for a straightforward pixel-to-pixel assignment between temporally corresponding superpixels. The changes in the superpixel shapes cause at least a relative pixel shift. Most often the superpixel shape deformation also changes the superpixel size, resulting in a superpixel pixel count difference between temporally adjacent superpixels. This requires an asymmetric pixel mapping instead of a one-to-one pixel mapping for the superpixel micro tracking.
The quality of superpixel micro tracking can be measured by their isogonic projection and the coverage reached for an asymmetric pixel mapping. The isogonic projection describes the relative pixel order given by the mapping, and the coverage refers to the percentage of pixels pointed at for the asymmetric mapping. For the case of a bad coverage the asymmetric mapping excludes large parts of the target superpixel by linking multiple and more than necessary pixels of the origin superpixel to a single pixel located within the target superpixel. This leads to unnecessary holes in the map, which exclude pixel locations of the target superpixel.
Apart from the temporal superpixel assignment aspect in image processing, similar problems arise for a multi-view superpixel assignment, which is required in light field camera and other multi-view applications. Temporal superpixels and multi-view superpixels are interchangeable items. Therefore, the temporal aspects of object related pixel assignments can be transferred to multi-view aspects of them. Those multi-view aspects are extensively used in image processing applied for light field cameras, for example.